Systems and methods to identify breaking application program interface changes

ABSTRACT

Systems and methods for managing Application Programming Interfaces (APIs) are disclosed. For example, the system may include one or more memory units storing instructions and one or more processors configured to execute the instructions to perform operations. The operations may include sending a first call to a first node-testing model associated with a first API and receiving a first model output comprising a first model result and a first model-result category. The operations may include identifying a second node-testing model associated with a second API and sending a second call to the second node testing model. The operations may include receiving a second model output comprising a second model result and a second model-result category. The operations may include performing at least one of sending a notification, generating an updated first node-testing model, generating an updated second node-testing model, generating an updated first call, or generating an updated second call.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/748,917, filed Jan. 22, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/457,548, filed Jun. 28, 2019, which is acontinuation of U.S. patent application Ser. No. 16/362,466, filed Mar.22, 2019, which claims the benefit of priority of U.S. ProvisionalApplication No. 62/694,968, filed Jul. 6, 2018. The disclosures of theaforementioned applications are expressly incorporated herein byreference in their entirety.

This application relates to U.S. patent application Ser. No. 16/151,385,filed Oct. 4, 2018, and titled “Data Model Generation Using GenerativeAdversarial Networks.” This application also relates to U.S. patentapplication Ser. No. 16/362,537, filed Mar. 22, 2019, and titled“Automated Honeypot Creation Within a Network.” The disclosures of theaforementioned applications are expressly incorporated herein byreference in their entirety.

BACKGROUND

Developers have a need to create and/or manage complicated systems thatinclude large numbers of Application Program Interfaces (APIs) thatreceive high call volume. For example, dozens, hundreds, or moreconnected APIs may receive millions, billions, or more calls per day.

In a system, an API call may result in API node output that is passed todownstream API nodes. Thus, a single API call may trigger dozens,hundreds, or even more downstream API calls. Often, the routing pathwaysthat API calls take through a set of nodes may be unknown or difficultto predict, owing to the complexity of the API system and thecustomizable nature of API calls. Further, node output from one API maybe incompatible with another API, resulting in errors, failures,breakpoints, and other problems that prevent a system from successfullyprocessing an API call. This creates challenges for developers, who maybe unable to develop a system that can handle some API calls that thesystem receives.

Developers have limited tools for testing APIs during production.Typically, testing consists of passing actual or synthetic API calls toAPI nodes to identify and predict possible break points. This method oftesting may be inefficient and consume large quantities of computingresources because APIs are often running. This method may also require ahigh degree of human oversight to generate and monitor tests. Further,problems may go unidentified and undetected until APIs are released andactual API calls trigger errors.

Therefore, in view of the shortcomings and problems with conventionalapproaches to training models, there is a need for efficient,unconventional systems that identify problems with API systems.

SUMMARY

The disclosed embodiments provide unconventional methods and systems formanaging APIs. The disclosed systems and methods may be implementedusing a combination of conventional hardware and software as well asspecialized hardware and software, such as a machine constructed and/orprogrammed specifically for performing functions associated with thedisclosed method steps.

Consistent with the present embodiments, there is provided a system formanaging APIs. The system may include one or more memory units storinginstructions and one or more processors configured to execute theinstructions to perform operations. The operations may include sending afirst call to a first node-testing model associated with a first API andreceiving from the first node-testing model, a first model output basedon the first call. The first model output may include a first modelresult and a first model-result category. The operations may includeidentifying a second node-testing model associated with a second API andsending a second call to the second node-testing model, the second callbeing based on the first model result. The operations may includereceiving, from the second node-testing model, a second model outputbased on the second call. The second model output may include a secondmodel result and a second model-result category. The operations mayinclude performing at least one of: sending a notification to a devicebased on at least one of the first model result, the first model-resultcategory, the second model result, or the second model-result category;generating an updated first node-testing model based on the firstnode-testing model; generating an updated second node-testing modelbased on the second node-testing model; generating an updated firstcall; or generating an updated second call.

Consistent with the present embodiments, a method for managing APIs isdisclosed. The method may include sending a first call to a firstnode-testing model associated with a first API and receiving from thefirst node-testing model, a first model output based on the first call.The first model output may include a first model result and a firstmodel-result category. The method may include identifying a secondnode-testing model associated with a second API and sending a secondcall to the second node-testing model, the second call being based onthe first model result. The method may include receiving, from thesecond node-testing model, a second model output based on the secondcall. The second model output may include a second model result and asecond model-result category. The method may include performing at leastone of: sending a notification to a device based on at least one of thefirst model result, the first model-result category, the second modelresult, or the second model-result category; generating an updated firstnode-testing model based on the first node-testing model; generating anupdated second node-testing model based on the second node-testingmodel; generating an updated first call; or generating an updated secondcall.

Consistent with other disclosed embodiments, non-transitory computerreadable storage media may store program instructions, which areexecuted by at least one processor device and perform any of the methodsdescribed herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several embodiments and, togetherwith the description, serve to explain the disclosed principles. In thedrawings:

FIG. 1 is a diagram of an exemplary system for managing APIs, consistentwith disclosed embodiments.

FIG. 2A is an illustration of an exemplary interface, consistent withdisclosed embodiments.

FIG. 2B is an illustration of an exemplary interface, consistent withdisclosed embodiments.

FIG. 3 depicts an exemplary API management system, consistent withdisclosed embodiments.

FIG. 4 depicts an exemplary process for training and implementing anode-testing model, consistent with disclosed embodiments.

FIG. 5 depicts an exemplary system for training a translation model,consistent with the disclosed embodiments.

FIG. 6 depicts an exemplary process for managing APIs, consistent withdisclosed embodiments.

FIG. 7 depicts an exemplary process for training a node-testing model,consistent with disclosed embodiments.

FIG. 8 depicts an exemplary process for testing a translation model,consistent with disclosed embodiments.

FIG. 9 depicts an exemplary process for implementing a translationmodel, consistent with disclosed embodiments.

FIG. 10 depicts an exemplary process for training and implementing anode-imitating model, consistent with disclosed embodiments.

FIG. 11, depicts an exemplary process for managing unauthorized APIcalls, consistent with disclosed embodiments.

FIG. 12 depicts an exemplary process for training a node-imitatingmodel, consistent with disclosed embodiments.

FIG. 13 depicts an exemplary process for identifying suspicious data,consistent with disclosed embodiments.

DESCRIPTION OF THE EMBODIMENTS

Disclosed embodiments improve API management by identifying andimproving API call errors, API routing, API versioning, and unauthorizedAPI access. APIs of the embodiments may include remote APIs (web-based,cloud-based, or server-based APIs) and/or local APIs (APIs hosted on alocal machine or local network). In the embodiments, APIs may includecommunication protocols and one or more software or programminglibraries. The embodiments provide unconventional systems and methodsfor training models to test API nodes, translate API calls between APIversions, and imitate API nodes. These systems and methods may be usedto facilitate effective communication between APIs of differentversions, that otherwise may be unable to effectively communicate. AnAPI version may be a set of rules or parameters associated with aparticular release date, which an API uses to operate. An API call maybe configured for one version of an API, and a second version of the APImay be unable to produce API output based on the API call or may producean error. Further, the embodiments may provide unconventional systemsand methods for profiling and clustering datasets, identifying datasetsimilarities, or identifying data lineage.

Systems and methods of disclosed embodiments may involve datasetscomprising actual data reflecting real-world conditions, events, ormeasurement. However, in some embodiments, disclosed systems and methodsmay fully or partially involve synthetic data (e.g., anonymized actualdata or fake data). Datasets of disclosed embodiments may have arespective data schema (i.e., structure), including a data type,key-value pair, label, metadata, field, relationship, view, index,package, procedure, function, trigger, sequence, synonym, link,directory, queue, or the like. Datasets of the embodiments may containforeign keys, i.e. data elements that appear in multiple datasets andmay be used to cross-reference data and determine relationships betweendatasets. Foreign keys may be unique (e.g., a personal identifier) orshared (e.g., a postal code). Datasets of the embodiments may be“clustered,” i.e., a group of datasets may share common features, suchas overlapping data, shared statistical properties). Clustered datasetsmay share hierarchical relationships (i.e., data lineage).

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings and disclosedherein. Wherever convenient, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts. Thedisclosed embodiments are described in sufficient detail to enable thoseskilled in the art to practice the disclosed embodiments. It is to beunderstood that other embodiments may be utilized and that changes maybe made without departing from the scope of the disclosed embodiments.Thus, the materials, methods, and examples are illustrative only and arenot intended to be necessarily limiting.

FIG. 1 is a diagram of exemplary system 100 to manage APIs, consistentwith disclosed embodiments. As shown, system 100 may include API systems102 a, 102 b, 102 n, an API management system 104, an interface 106, amodel storage 108, a database 110, a client device 112, and anenforcement system 114. Components of system 100 may be connected toeach other through a network 114.

In some embodiments, aspects of system 100 may be implemented on one ormore cloud services designed to generate (“spin-up”) one or moreephemeral container instances in response to event triggers, assign oneor more tasks to a container instance, and terminate (“spin-down”) acontainer instance upon completion of a task. By implementing methodsusing cloud services, disclosed systems efficiently provision resourcesbased on demand and provide security advantages because the ephemeralcontainer instances may be closed and destroyed upon completion of atask. That is, the container instances do not permit access from outsideusing terminals or remote shell tools like SSH, RTP, FTP, or CURL, forexample. Further, terminating container instances may include destroyingdata, thereby protecting sensitive data. Destroying data can providesecurity advantages because it may involve permanently deleting data(e.g., overwriting data) and associated file pointers.

As will be appreciated by one skilled in the art, the components ofsystem 100 can be arranged in various ways and implemented with anysuitable combination of hardware, firmware, and/or software, asapplicable. For example, as compared to the depiction in FIG. 1, system100 may include a larger or smaller number of client devices,interfaces, model optimizers, model storages, and databases. Inaddition, system 100 may further include other components or devices notdepicted that perform or assist in the performance of one or moreprocesses, consistent with the disclosed embodiments. The exemplarycomponents and arrangements shown in FIG. 1 are not intended to limitthe disclosed embodiments.

API systems 102 a, 102 b, 102 n may include remote APIs (web-based,cloud-based, or server-based APIs) connected to system 100 via one ormore networks (e.g., via network 116) and/or local APIs (APIs hosted ona local machine or local network of system 100). API systems may includea function, a microservice, subroutine, or another component of an APIsystem. API systems 102 a, 102 b, 102 n may include communicationprotocols and one or more software or programming libraries. API systems102 a, 102 b, 102 n are configured to receive input (API calls) andreturn API output in response to the calls. In some embodiments, one ormore of API systems 102 a, 102 b, 102 n are stand-alone API platformsthat may include a plurality of subroutines handled by a plurality ofAPI nodes. In some embodiments, two or more API systems 102 a, 102 b,102 n are components of the same API platform and operate as API nodes.In some embodiments, API systems 102 a, 102 b, 102 n are configured tosend calls to other API systems or nodes of system 100 and receive APIoutput in response (e.g., API system 102 a may send a call to API system102 b and receive a response in return). Calls between API systems 102a, 102 b, 102 n and calls within API systems 102 a, 102 b, 102 n may berouted by API management system 104.

API management system 104 may include one or more computing systemsconfigured to manage training of models for system 100 and route APIcalls, consistent with disclosed embodiments. API management system 104can be configured to receive API calls, models, and/or datasets fromother components of system 100 or other components not shown (e.g., viainterface 106). API management system 104 may be configured to train andimplement models, including machine learning models. API managementsystem 104 may be configured to generate models. In some embodiments,API management system 104 is configured to export models to othercomponents of system 100 and/or to external systems or devices (e.g.,client device 112). API management system 104 is disclosed in greaterdetail, below (in reference to FIG. 3).

Interface 106 can be configured to manage interactions between system100 and other systems using network 116. In some aspects, interface 106can be configured to publish data received from other components ofsystem 100. This data can be published in a publication and subscriptionframework (e.g., using APACHE KAFKA), through a network socket, inresponse to queries from other systems, or using other known methods.The data can be synthetic data, as described herein. As an additionalexample, interface 106 can be configured to provide information receivedfrom model storage 108 regarding available datasets. In various aspects,interface 106 can be configured to provide data or instructions receivedfrom other systems to components of system 100. For example, interface106 can be configured to receive instructions for generating data models(e.g., type of data model, data model parameters, training dataindicators, training hyperparameters, or the like) from another systemand provide this information to model optimizer 104. As an additionalexample, interface 106 can be configured to receive data includingsensitive portions from another system (e.g. in a file, a message in apublication and subscription framework, a network socket, or the like)and provide that components of system 100.

System 100 may include model storage 108. In some embodiments, some orall components of model storage 108 may be hosted on one or moreservers, one or more clusters of servers, or one or more cloud services.Model storage 108 may be connected to network 116 and may additionallybe directly connected to API management system 104 (connection notshown). In some embodiments, model storage 108 is a component of APImanagement system 104 or client device 112 (not shown).

Model storage 108 can include one or more databases configured to storedata models (e.g., machine-learning models or statistical models) anddescriptive information of the data models. Model storage 108 can beconfigured to provide information regarding available data models to auser or another system. The databases can include cloud-based databases,cloud-based buckets, or on-premises databases. The information caninclude model information, such as the type and/or purpose of the modeland any measures of classification error. Model storage 108 can includeone or more databases configured to store indexed and clustered modelsfor use by system 100. For example, model storage 108 may store modelsassociated with generalized representations of those models (e.g.,neural network architectures stored in TENSORFLOW or other standardizedformats). The databases can include cloud-based databases (e.g., AMAZONWEB SERVICES S3 buckets) or on-premises databases.

Database 110 can include one or more databases configured to store datafor use by system 100. The databases can include cloud-based databases(e.g., AMAZON WEB SERVICES S3 buckets) or on-premises databases.Database 110 can include one or more databases configured to storeindexed and clustered models for use by system 100, as described above.

Client device 112 may include one or more memory units and one or moreprocessors configured to perform operations consistent with disclosedembodiments. In some embodiments, client device 112 may includehardware, software, and/or firmware modules. Client device 112 may be aterminal, a kiosk, a mobile device, a tablet, a personal computer, aserver, a server cluster, a cloud service, a storage device, or aspecialized device configured to perform methods according to disclosedembodiments, or the like.

Enforcement system 114 may include a server including one or more memoryunits and one or more processors configured to perform operationsconsistent with disclosed embodiments. In some embodiments, enforcementsystem 114 may be configured to track and/or report malicious networkactivity. For example, enforcement system 114 may report to or be acomponent of a government agency (e.g., a cybercrimes agency) and/or anorganization that tracks malware, hackers, or the like. In someembodiments, enforcement system 114 may be a component of API managementsystem 104 (not shown). In some embodiments, enforcement system 114 maybe managed by a government agency, a nonprofit agency, a privateorganization, or another organization.

Network 116 may be a public network or private network and may include,for example, a wired or wireless network, including, without limitation,a Local Area Network, a Wide Area Network, a Metropolitan Area Network,an IEEE 1002.11 wireless network (e.g., “Wi-Fi”), a network of networks(e.g., the Internet), a land-line telephone network, or the like.Network 116 may be connected to other networks, not depicted, to connectthe various system components to each other and/or to external systemsor devices. In some embodiments, network 116 may be a secure network andrequire a password to access the network.

FIG. 2A is an illustration of exemplary interface 200, consistent withdisclosed embodiments. In some embodiments, the interface is displayedat one of interface 106, API management system 104, client device 112,or other component of system 100.

As shown, interface 200 may include a graphical representation of a deeplearning platform node-testing model 202. A node-testing model may be amodel that simulates (imitates) a corresponding API node by producingmodel output that simulates API output in response to an API call,consistent with disclosed embodiments. In some embodiments, anode-testing model may be a machine learning model, such as asequence-to-sequence model (seq2seq), which may be implemented using oneor more of a recurrent neural network mode (RNN), a long-short termmemory (LSTM) model, convolutional neural network (CNN), or anotherneural network model. In some embodiments, the node testing model mayinclude a synthetic data generation model or may be configured toimplement a synthetic data generation model. For example, a generativeadversarial network (GAN), a variational auto encoder, or any of theother neural networks previously mentioned may be implemented togenerate synthetic data. In some embodiments, a node testing model istrained using API output data and call data to simulate API output. Calldata may include actual or simulated API calls. An API node may be anAPI system (e.g., one of API system 102 a, 102 b, 102 n) or a componentof an API system (e.g., a function, a microservice, subroutine, or othercomponent of an API system).

In some embodiments, deep learning platform node-testing model 202 mayinclude a plurality of component node-testing models, includinginfrastructure node-testing model 204 a, data access node-testing model204 b, model monitoring node-testing model 204 c, launch node-testingmodel 206 a infrastructure monitoring node-testing model 206 b, Securedata-access node-testing model 206 c, data movement node-testing model206 d, model monitoring library node-testing model 206 e, stopnode-testing model 208 a secure data-movement node-testing model 208 b,and/or model distribution node-testing model 208 c. As shown, interface200 may depict deep learning platform node-testing model 202 and thecomponent node-testing models as discs. Node-testing models of FIG. 2Aare depicted for purposes of illustration only, and, as one of skill inthe art will appreciate, interface 200 may include additional, fewer, ordifferent node-testing models.

In some embodiments, deep learning platform node-testing model 202 andthe component node-testing models are configured to receive calls andreturn model output, consistent with disclosed embodiments. As shown inFIG. 2A, the component node-testing models may be presented as anunstructured system, with no known relationships between them. That is,routing pathways that calls may take as they are passed from deeplearning platform node-testing model 202 to component node-testingmodels may be unknown. In addition, although node-testing modelsdepicted in FIG. 2A have labels that may describe a function that thecorresponding API node is configured to perform (e.g., “securedata-movement”), user-interface 200 may display node-testing models asunlabeled node-testing models or as labels that do not describe afunction (e.g., “model 1”, “model A”, etc.).

FIG. 2B is an illustration of exemplary interface 250, consistent withdisclosed embodiments. In some embodiments, the interface is displayedat one of interface 106, API management system 104, client device 112,or other component of system 100. As shown, interface 200 may include agraphical representation of a deep learning platform node-testing model202. Interface 250 may display a structured arrangement of componentnode-testing models as described above in relation to interface 200. Inthe example of interface 250, component node-testing models are arrangedaccording to routing pathways and routing layers, with arrowsrepresenting routing pathways of calls and model output to downstreamnode-testing models. As one of skill in the art will appreciate,interface 200 may include additional, fewer, or different node-testingmodels, routing layers, and/or routing pathways.

As shown in FIG. 2B, a call to deep learning platform node-testing model202 may produce one or more node-testing model outputs that are routeddownstream to destination node-testing models in a routing layer 1 thatincludes the node-testing models including infrastructure node-testingmodel 204 a, data access node-testing model 204 b, and model monitoringnode-testing model 204 c. Node-testing models of routing layer 1 mayproduce one or more model output and pass the model outputs to one ormore node-testing models of a routing layer 2. For example,infrastructure node-testing model 204 a may produce model output that ispassed to launch node-testing model 206 a and to infrastructuremonitoring node-testing model 206 b. Similarly, node-testing models ofrouting layer 2 may produce one or more model outputs and pass the modeloutputs to one or more node-testing models of a routing layer 3. Forexample, infrastructure monitoring node-testing model 206 b may pass amodel output to stop node-testing model 208 a.

Interface 250 may display routing pathways and routing layers asdetermined by, for example, a routing table and/or a routing model,consistent with disclosed embodiments. For example, a routing table mayinclude a list of calls and corresponding destination API nodes (or APInode-testing models). A routing model may be trained on call data andAPI node output data to predict API node destinations, consistent withdisclosed embodiments.

Interface 250 may be configured to display a model-result categoryassociated with the node-testing model output. A model-result categorymay include a confidence level that a model result matches an APIresult. A model-result category may indicate a success, a warning, orfailure of the call. In exemplary interface 250, model-result categoriesinclude “high confidence of working,” “possible failure,” “highconfidence of failure,” and “unknown.” For example, “high confidence ofworking” may indicate a high likelihood that the node testing-modelproduces a model output which does not contain an error or warning.“Possible failure” may indicate low likelihood in the model output ormay indicate that the model output may contain an error or warning.“High confidence of failure” may indicate high likelihood that the modeloutput contains an error or warning. “Unknown” may indicate that themodel is not configured to produce model output for a given call, thatthe likelihood of any particular model result is below a predeterminedthreshold, and/or that the result has an invalid or unknown data schema.

As described, interface 250 displays routing pathways, model-resultcategories. Accordingly, interface 250 identifies break points, errors,and can be used for managing APIs.

FIG. 3 is an illustration of an exemplary API management system 104,consistent with disclosed embodiments. As shown, API management system104 includes one or more processors 310, one or more I/O devices 320,and one or more memory units 330. In some embodiments, some or allcomponents of API management system 104 may be hosted on a device, acomputer, a server, a cluster of servers, or a cloud service. In someembodiments, API management system 104 is a scalable system configuredto efficiently manage resources and enhance security by provisioningcomputing resources in response to triggering events and terminatingresources after completing a task (e.g., a scalable cloud service thatspins up and terminates container instances).

As depicted in FIG. 3, API management system 104 may include one or moreprocessors 310, input/output units (I/O devices) 320, and one or morememory units 330. FIG. 3 is an exemplary configuration of API managementsystem 104. As will be appreciated by one skilled in the art, thecomponents and arrangement of components included in API managementsystem 104 may vary. For example, as compared to the depiction in FIG.3, API management system 104 may include a larger or smaller number ofprocessors 310, I/O devices 320, or memory units 330. In addition, APImanagement system 104 may further include other components or devicesnot depicted that perform or assist in the performance of one or moreprocesses consistent with the disclosed embodiments. The components andarrangements shown in FIG. 3 are not intended to limit the disclosedembodiments, as the components used to implement the disclosed processesand features may vary.

Processor 310 may be known computing processors, including amicroprocessor. Processor 310 may constitute a single-core ormultiple-core processor that executes parallel processes simultaneously.For example, processor 310 may be a single-core processor configuredwith virtual processing technologies. In some embodiments, processor 310may use logical processors to simultaneously execute and controlmultiple processes. Processor 310 may implement virtual machinetechnologies, or other known technologies to provide the ability toexecute, control, run, manipulate, store, etc., multiple softwareprocesses, applications, programs, etc. In another embodiment, processor310 may include a multiple-core processor arrangement (e.g., dual core,quad core, etc.) configured to provide parallel processingfunctionalities to allow execution of multiple processes simultaneously.One of ordinary skill in the art would understand that other types ofprocessor arrangements could be implemented that provide for thecapabilities disclosed herein. The disclosed embodiments are not limitedto any type of processor(s) 310. Processor 310 may execute variousinstructions stored in memory 330 to perform various functions of thedisclosed embodiments described in greater detail below. Processor 310is configured to execute functions written in one or more knownprogramming languages.

I/O devices 320 may include at least one of a display, an LED, a router,a touchscreen, a keyboard, a microphone, a speaker, a haptic device, acamera, a button, a dial, a switch, a knob, a transceiver, an inputdevice, an output device, or another I/O device to perform methods ofthe disclosed embodiments. I/O devices 320 may be components of aninterface of API management system 104 (e.g., an interface such asinterface 106).

Referring again to FIG. 3, memory 330 may be a volatile or non-volatile,magnetic, semiconductor, optical, removable, non-removable, or othertype of storage device or tangible (i.e., non-transitory)computer-readable medium, consistent with disclosed embodiments. Asshown, memory 330 may include data 331, including of at least one ofencrypted data or unencrypted data. Data 331 may include one or moremodel indexes, model parameters, model hyperparameters, model codes,dataset indexes, API routing tables, and/or datasets, consistent withdisclosed embodiments.

Programs 335 may include one or more programs (e.g., modules, code,scripts, or functions) used to perform methods consistent with disclosedembodiments. Programs may include operating systems (not shown) thatperform known operating system functions when executed by one or moreprocessors. Disclosed embodiments may operate and function with computersystems running any type of operating system. Programs 335 may bewritten in one or more programming or scripting languages. One or moreof such software sections or modules of memory 330 can be integratedinto a computer system, non-transitory computer-readable media, orexisting communications software. Programs 335 can also be implementedor replicated as firmware or circuit logic.

Programs 335 may include a model-training module 336, a node-testingmodule 337, a translation module 338, a routing module 339, a datasetclustering module 340, honeypot module 341, and/or other modules notdepicted to perform methods of the disclosed embodiments. In someembodiments, modules of programs 335 may be configured to generate(“spin up”) one or more ephemeral container instances to perform a taskand/or to assign a task to a running (warm) container instance,consistent with disclosed embodiments. Modules of programs 335 may beconfigured to receive, retrieve, and/or generate models, consistent withdisclosed embodiments. Modules of programs 335 may be configured toreceive, retrieve, and/or generate datasets (e.g., to generate syntheticdatasets, data samples, or other datasets), consistent with disclosedembodiments. Modules of programs 335 may be configured to performoperations in coordination with one another. For example, routing module339 may send a model training request to model-training module 336 andreceive a trained model in return, consistent with disclosedembodiments.

Model-training module 336 may be configured to train one or more modelsand/or perform hyperparameter tuning of one or more models, includingmachine learning models. For example, model-training module 336 can beconfigured to receive input of one or more thresholds, one or more lossfunctions, and/or one or more limits on a number of interactions andapply the input for optimizing a received model and/or correspondinggeneralized representation such as a neural network. In someembodiments, training of a model terminates when a training criterion issatisfied. In some embodiments, model-training module 336 is configuredto adjust model parameters during training. The model parameters mayinclude weights, coefficients, offsets, or the like. Training can besupervised or unsupervised.

Model-training module 336 can be configured to select or generate modelparameters (e.g., number of layers for a neural network, kernel functionfor a kernel density estimator, or the like), update traininghyperparameters, and evaluate model characteristics. For example, modelcharacteristics may include a model type (e.g., an RNN, a convolutionalneural network (CNN), a random forest, or another model type), a modelparameter, a model hyperparameter (including training a hyperparameterand/or an architectural hyperparameter), a desired outcome,belongingness to a model cluster, and/or belonginess of a model trainingdataset to a dataset cluster, the similarity of synthetic data generatedby a model to actual data, or other characteristics.

In some embodiments, model-training module 336 is configured to performa search of a hyperparameter space and select new hyperparameters. Thissearch may or may not depend on the values of a performance metricobtained for other trained models. In some aspects, model-trainingmodule 336 can be configured to perform a grid search or a randomsearch. The hyperparameters can include training hyperparameters, whichcan affect how training of the model occurs, or architecturalhyperparameters, which can affect the structure of the model.

Consistent with disclosed embodiments, hyperparameters can includetraining hyperparameters such as learning rate, batch size, number oftraining batches, number of epochs, chunk size, time window, input noisedimension, or the like. Hyperparameters can include architecturalparameters such as number of layers in a neural network, the choice ofactivation function for a neural network node, the layers in a CNN orthe like. For example, when the stored model comprises a generativeadversarial network (CAN), training hyperparameters for the model caninclude a weight for a loss function penalty term that penalizes thegeneration of training data according to a similarity metric. As afurther example, when the stored model comprises a neural network, thetraining hyperparameters can include a learning rate for the neuralnetwork. As an additional example, when the model is a CNN,architectural hyperparameters can include the number and type of layersin the convolutional neural network.

In some embodiments, model-training module 336 includes programs toapply one or more templates to a data model (e.g., a model retrievedfrom model storage 108) and apply the templates to generate ageneralized representation of the retrieved model (e.g., a neuralnetwork). Model-training module 336 may include programs to providetrained generalized representations to model storage 108 for storing inassociation with corresponding models.

Node-testing module 337 may be configured to train models to test and/orimitate API nodes, consistent with disclosed embodiments. Node-testingmodule 337 may include or be configured to train and implement models incoordination with or independent from other modules of programs 335,consistent with disclosed embodiments. For example, node-testing module337 may include or be configured to implement one or more node-testingmodels and/or one or more node-imitating models, consistent withdisclosed embodiments. Node-testing models and/or node-imitating modelsmay include machine learning models. In some embodiments, the machinelearning models may include an RNN, a long-short term memory (LSTM)model, convolutional neural network (CNN), a seq2seq model, generativeadversarial network (GAN), an autoencoder, a variational autoencoder, oranother neural network model. The machine learning models may be trainedusing API call data (e.g., API calls, API outputs, and/or APIidentifiers) to receive API calls and generate model outputs thatinclude model results match API call results. API identifiers mayinclude a date or version number associated with an API, a source IPaddress, a destination IP address, and/or a schema to which an API callconforms. The model result may include a message (e.g., an error messageor a warning message). In some embodiments, the models may be trained togenerate model outputs that include one or more model-result categories.A model-result category may include a confidence level that a modelresult matches an API result. A model-result category may indicate asuccess, a warning, or failure of the call. In some embodiments, themachine learning models (e.g., node-testing models or node-imitatingmodels) may be configured to retrieve API call data from a storage(e.g., data 331, database 110, or other data storage) for modeltraining, API testing, or call processing. In some embodiments, themachine learning models may be configured to receive API call data asAPI nodes calls are received and processed by, for example, one or moreAPI systems (e.g., API systems 102 a, 102 b, 102 c). Node-testing module337 and models of node-testing module 337 may be configured to receiveand transmit data, including API call data, from and to other componentsof system 100 or from computing components outside system 100 (e.g., viainterface 106).

Translation module 338 may be configured to translate a call (an inputcall) to a different call (i.e., a translated call). The input call maybe associated with an API and/or a particular version of an API.Translation module 338 may be configured to translate any number ofkinds of inputs. In some embodiments, the inputs to translation module338 may be model outputs or API outputs, such as an API responseproduced by an API. The translated inputs may therefore also betranslated outputs. These translated outputs may also be associated withan API and/or a particular version of an API. In some embodiments,translation module 338 may perform multiple translations on the sameinput.

In some embodiments, translation module 338 may be configured to receivean input from another module, such as routing module 339, another partof API Management System 104, or another device connected to ManagementSystem 104. This input may be an API call, API dataset, API response, orother code. Translation module 338 may determine certain characteristicsabout the input, such as an API version it corresponds to, a schema itfollows, or a source IP address or destination IP address associatedwith it.

Translation module 338 may be configured to translate an input based onits characteristics. For example, if translation module 338 determinesthat the input is associated with a first version of an API, it maytranslate the input to a different input based on that version of anAPI. In some embodiments, translation module 338 may be configured totranslate an input based on a first API node to which the input is beingsent, and/or a second API node from which the input is sent. Thistranslation may be based on characteristics of the first and/or secondAPI node, such as a version of API running on the first and/or secondAPI node, an identifier associated with the first and/or second APInode, or attributes of the surroundings of the first and/or second APInode. For example, if an input is being sent to a first node running afirst version of an API, such as a newer or older version of an APIrelative to the version associated with the input, translation module338 may translate the input into an input associated or compliant withthe version of an API running on the first node. In this example, thefirst node may be called a destination API node, as it is thedestination of a translated input or output.

In some embodiments, translation module 338 may be configured totranslate an input, such as an API call, to a different input, withoutthe use of a library or stored versions of an API. For example,translation module 338 may be configured by another module, such asmodel-training module 336, which may be a machine-learning model and/ora neural network. Machine-learning models of translation module 338 mayinclude an RNN model. Translation module 338 may operate according torules, which may be pre-programmed rules, rules learned throughmachine-learning, or a combination of both. In some embodiments,translation module 338 may be configured to generate, train, and/orimplement translation models, which may be used to translate inputs.

Translation module 338 may also be configured to send a translated inputto another module, such as routing module 339, another part of APIManagement System 104, or another device connected to Management System104. In some embodiments, translation module 338 may send a translatedinput to another module for further processing. For example, translationmodule 338 may send a translated input to another translation module forfurther translation. In some embodiments, translation module 338 maysend a translated input to a module or device that processes the inputto produce a result. For example, translation module 338 may send an APIcall to an API that processes the API call to produce an API resultresponsive to the call.

Translation module 338 may also be configured to generate, train, and/orimplement versioning models, which may be configured to determine an APIassociated with an input (e.g., an API call), consistent with disclosedembodiments. In some embodiments, the versioning model may be configuredto associate an API version with a corresponding model (e.g., atranslation model, a node-testing model, and/or a node-imitating model).In some embodiments, versioning models may be rule-based. For example, aversioning model may be configured to apply a table (e.g., lookup table)to determine an API version associated based on information associatedwith an API call (e.g., metadata, a header, an API identifier). In someembodiments, versioning models may be machine learning models trained todetermine an API version using API call data, consistent with disclosedembodiments. For example, the versioning model may be trained toidentify an API version based on data contained within or associatedwith an API call (e.g., based on API call syntax, commands (e.g.,function calls), a data schema of a dataset, or other data).

Routing module 339 may be configured to identify routing pathways and/orto route calls between API nodes, consistent with disclosed embodiments.In some embodiments, routing module 339 is configured to receive arouting table from another component of system 100 or a computingcomponent outside system 100 (e.g., via interface 106). In someembodiments, routing module 339 is configured to generate, train, and/orimplement a versioning model, as previously described. In someembodiments, routing module 339 may be configured to retrieve a routingtable from a storage (e.g., data 331, database 110, or other datastorage). A routing table may include information specifying where toroute an API call and/or where to route an API output. The routing tablemay include identifiers specifying API nodes. For example, the routingtable may include information specifying that an API call (or APIoutput) containing a first identifier should be routed to a first APInode. The identifier may be an API address (IP address, domain name, orother address), an API name, an API version, an API function, or otheridentifier.

In some embodiments, routing module 339 may be configured to generate arouting table, consistent with disclosed embodiments. For example,routing module 339 may generate a routing table by extractinginformation in an API call specifying a routing pathway and generatingor updating a routing table based on the extracted information. In someembodiments, routing module 339 may generate a routing table by creatinga graph based on routing logs between nodes. In some embodiments,routing module 339 may be configured to train and implement a routingmodel. The routing model may include one or more machine-learningmodels. In some embodiments, the machine-learning models may include anRNN, a long-short term memory (LSTM) model, or another neural networkmodel. The machine-learning models may be trained using API call data(e.g., API calls and/or API outputs) to predict a routing pathway basedon the API call data. In some embodiments, the machine-learning modelsmay be configured to retrieve API call data from a storage (e.g., data331, database 110, or other data storage) for model training, APItesting, or call processing. In some embodiments, the machine-learningmodels may be configured to receive API call data in real-time as APInodes calls are received and processed by, for example, one or more APIsystems (e.g., API systems 102 a, 102 b, 102 c). Routing module 339 maybe configured to receive and transmit data, including API call data,from and to other components of system 100 or from components outsidesystem 100 (e.g., via interface 106).

Dataset-clustering module 340 may be configured to group, or “cluster,”datasets, consistent with disclosed embodiments. Dataset-clusteringmodule 340 may include or be configured to implement one or moredata-profiling models. A data-profiling model may includemachine-learning models and statistical models to determine the dataschema and/or a statistical profile of a dataset (i.e., to profile adataset), consistent with disclosed embodiments. The data-profilingmodel may include an RNN model, a CNN model, or other machine-learningmodel. The data-profiling model may include algorithms to determine adata type, key-value pairs, row-column data structure, statisticaldistributions of information such as keys or values, or other propertyof a data schema. The data-profiling model may be configured toimplement univariate and multivariate statistical methods. Thedata-profiling model may include a regression model, a Bayesian model, astatistical model, a linear discriminant analysis model, or otherclassification model configured to determine one or more descriptivemetrics of a dataset. For example, the data-profiling model may includealgorithms to determine an average, a mean, a standard deviation, aquantile, a quartile, a probability distribution function, a range, amoment, a variance, a covariance, a covariance matrix, a dimensionand/or dimensional relationship (e.g., as produced by dimensionalanalysis such as length, time, mass, etc.) or any other descriptivemetric of a dataset.

In some embodiments, the data-profiling models may be configured toreturn a statistical profile of a dataset. The statistical profile mayinclude a plurality of descriptive metrics. For example, the statisticalprofile may include an average, a mean, a standard deviation, a range, amoment, a variance, a covariance, a covariance matrix or any otherstatistical metric of the selected dataset. In some embodiments, thestatistical metric may be a similarity metric representing a measure ofsimilarity between data in a dataset. The similarity metric may be basedon a covariance matrix, a variance, a frequency of overlapping values,or other measure of statistical similarity.

Dataset-clustering module 340 may be configured to generate or implementa data-mapping model. A data-mapping model may include machine-learningmodels to generate edges between nodes, the nodes being datasets (i.e.,cluster datasets, data mapping or data crawling). The data-mapping modelmay include at least one of an RNN model, a CNN model, a random forestmodel, a bag-of-words model, a multilayer perceptron model, a gatedrecurrent unit model, a seq2seq model, or other machine-learning model.An edge may comprise an indicator of a hierarchical relationship (e.g.,a data lineage, parent-child relationship, derived data, an ambiguoushierarchy), and may include an overlap score indicating the amount ofoverlap between datasets.

In some embodiments, dataset-clustering module 340 may be configured togenerate or retrieve a data-mapping model from a data storage (e.g.,model storage 108). Dataset-clustering module 340 may identify andretrieve a data-mapping model based on a statistical profile, a dataschema, a model index, and/or a model search strategy. Consistent withdisclosed embodiments, the search strategy may include a random searchor a grid search. In some embodiments, dataset-clustering module 340 maybe configured to receive a plurality of datasets and retrieve adata-mapping model previously used to generate edge data for one of thereceived datasets. Dataset-clustering module 340 may be configured toretrieve a data-mapping module previously used for a dataset that sharefeatures of a data schema of one of the received datasets. In someaspects, dataset-clustering module 340 may be capable of retrieving adata-mapping model used for a dataset having a statistical similaritymetric with one of the received datasets that meets a thresholdcriterion.

In some embodiments, the data-mapping model includes machine-learningmodels or other models to identify foreign keys and maintain an index offoreign keys (e.g., a data crawler model). The foreign keys may beunique or shared, consistent with disclosed embodiments. The foreignkeys may be stored in, for example, foreign database 110 and/or data331. The data-mapping model may be configured to predict foreign keys byidentifying candidate foreign keys and determining a foreign key scorebased on at least one of an index of foreign keys or a search of adataset. For example, the data-mapping model may be configured todetermine a foreign key score based on a frequency of occurrence of acandidate foreign key in one or more datasets or based on a labelassociated with the candidate foreign key. As another example, thedata-mapping model may be capable of assigning a foreign key score to acandidate foreign key based on its occurrence in a data column thatincludes known foreign keys.

The data-mapping model of dataset-clustering module 340 may beconfigured to connect datasets (i.e., generate edges between datasets)based on at least one of a foreign key, a data schema, or a similaritymetric. Edge data may include information indicating a similaritybetween datasets (e.g., a measure of data overlap, correlation,covariance, or other measure of statistical similarity) or ahierarchical relationship (e.g., derived data, parent-childrelationships). The data-mapping model may be configured to receive aplurality of datasets and generate edges based solely on the receiveddatasets. In some embodiments, the data-mapping model may be configuredto receive a plurality of datasets and generate edges based on thereceived datasets and on stored, clustered datasets.

Clustering module 340 may include or be configured to implement adata-classification model. The data-classification model may includemachine-learning models to classify datasets based on the data schema,statistical profile, foreign keys, and/or edges. The data-classificationmodel may be configured to segment datasets, consistent with disclosedembodiments. Segmenting may include classifying some or all data withina dataset, marking or labeling data (e.g., as duplicate), cleaning adataset, formatting a dataset, or eliminating some or all data within adataset based on classification. The models may be configured toclassify data elements as actual data, synthetic data, relevant data foran analysis goal or topic, data derived from another dataset, or anyother data category. The data-classification model may include a CNN, arandom forest model, an RNN model, a support vector machine model, oranother machine-learning model.

Honeypot module 341 may be configured to implement a honeypot computersecurity mechanism. In some embodiments, honeypot module 341 may beconfigured to automatically generate and implement honeypots in responseto detected malicious actors. That is, honeypot module 341 may beconfigured to detect an unauthorized API call and provide, in response,synthetic data that appears to be real API output. After an unauthorizedcall is detected, API management system 104 may continue to routelegitimate or authorized calls to API nodes, or to generate new APInodes to manage legitimate calls (e.g., a secondary live system).Honeypot module 341 may include or be configured to train and implementmodels in coordination with or independent from other modules ofprograms 335, consistent with disclosed embodiments.

In some embodiments, honeypot module 341 is configured to receive,retrieve, and/or generate a node-imitating model, consistent withdisclosed embodiments. Model output of the node-imitating model mayappear to be real API output to an unauthorized user (i.e., the modelmay be a component of a honeypot computer-security system mechanism).Generating and/or training a node-imitating model may be based on datareceived or retrieved from a model storage, including one or moremodels, model characteristics, and/or training criteria. Anode-imitating model may be configured to retrieve and/or receive data(e.g., from database 110), including one or more datasets. In someembodiments, training a node-imitating model may include receiving anode-testing model from a data storage (e.g., model storage 108) andtraining the node-testing model.

Honeypot module 341 may train a node-imitating model to generate modeloutput based on API output. In some embodiments, the API managementsystem 104 trains node-imitating model 1022 until one or more trainingcriteria are satisfied, consistent with disclosed embodiments.

In some embodiments, training a node-imitating model to generate modeloutput includes training the model to generate synthetic data. Forexample, a node-imitating model may be trained to generate syntheticdata that has a data profile that, when compared to API output data,satisfies a similarity metric, consistent with disclosed embodiments. Insome embodiments, generating synthetic data includes adjusting APIoutput by adding noise. In some embodiments, generating synthetic dataincludes altering a data schema of API output data (e.g., switching datalabels or otherwise relabeling data). In some embodiments, generatingsynthetic data includes replacing data in API output with syntheticdata. For example, a node-imitating model may be trained to identifysensitive information (e.g., account numbers, social security numbers,names, addresses, API keys, network or IP addresses, or the like) andreplace sensitive information with non-sensitive information havingsimilar properties (e.g., identifying a 16-digit account number andreplacing it with a fake 16-digit account number).

In some embodiments, model output of a node-imitating model may includea data marker. The data marker may include one or more data elements(e.g. a string or a number). A data marker may be randomly generated. Adata marker may include a Universally Unique Identifier (UUID). In someembodiments, honeypot module 341 is configured to store a data marker.In some embodiments, the data marker is associated with an API call, alocation, an API node, a node-testing model (e.g., data marker may bestored in a relational database). In some embodiments, honeypot module341 is configured to receive a dataset and determine whether the datasetincludes a data marker. Determine whether the dataset includes a datamarker may include comparing the dataset to a stored data marker.

In some embodiments, API management system 104 may train anode-imitating model to generate model output having a predetermineddata profile. For example, the predetermined data profile may be basedon one or more data markers (e.g., the node-imitating model may generatedata with two data markers having a predetermined covariance betweendata columns). The predetermined data profile may be unique to the modeloutput, such that the appearance of the predetermined data profileconfirms that the data to which the profile is attached is the modeloutput. In some embodiments, the predetermined data profile may usetoken keys to determine whether received data matches a previous modeloutput. API management system 104 may retain token keys to confirm aninstance of model output.

In some embodiments, honeypot module 341 is configured to determine asuspiciousness score of the data. In some embodiments, thesuspiciousness score is a likelihood that data derives from API calls toan API node (e.g. API systems 102 a, 102 b, 102 n or their components)and/or API management system 104. In some embodiments, thesuspiciousness score may be a number (e.g., 0-100) or a category (e.g.,highly, medium, low). In some embodiments, the suspiciousness score isbased on detecting a data marker. For example, the suspiciousness scoremay be based on a frequency of a data marker in a dataset.

In some embodiments, honeypot module 341 is configured to determine asuspiciousness score based on a data profile. For example, honeypotmodule 341 may determine a similarity metric of a data profile and adata profile of a stored dataset (or a stored data profile). Thesuspiciousness score may be based on the similarity metric (e.g., if thesimilarity metric indicates high similarity, the suspiciousness scoremay be high).

In some embodiments, honeypot module 341 may be configured to identifyunauthorized API calls (e.g., calls from an intruder, a hacker, amalicious actor internal or external to a network). In some embodiments,the unauthorized call may be associated with an authorized user of acomputer network (e.g., a malicious actor may use an authorized user'saccount credentials; an employee may access a computer system forillegitimate purposes, etc.). Identifying unauthorized API calls mayinclude classifying API calls based on at least one of a user account, alog, a failed authentication attempt, a packet sniffing event, a rate ofpinging, an Internet Protocol (IP) address, or a media access control(MAC) address.

Honeypot module 341 may include or be configured to train and implementa call-classification model to classify a call as an unauthorized call,consistent with disclosed embodiments. The call-classification model mayinclude a machine-learning model, including a decision tree, an RNN, aCNN, a multilayer perceptron (MLP), or another machine-learning model.The call-classification model may include rule-based models (e.g., amodel to detect calls from an unknown location).

For example, honeypot module 341 may be configured to train acall-classification model to detect a suspicious call or suspicious callpattern that differs from normal API call traffic. Honeypot module 341may be configured to detect a suspicious call. In some embodiments,honeypot module 341 may be configured to identify a location associatedwith the suspicious call (e.g., an account, an Internet Protocol (IP)address, a media access control (MAC) address, or a uniform resourcelocator (URL)). In some embodiments, honeypot module 341 may beconfigured to route a suspicious call or calls from a locationassociated with the suspicious call to a node-imitating model and/orblock calls from a location associated with the suspicious call.

Honeypot module 341 may be configured to determine a call relates to amalicious campaign, consistent with disclosed embodiments. For example,the call may originate from or be associated with a location of asuspicious call or an unauthorized call. The call may have callcharacteristics that are similar to a suspicious call or an unauthorizedcall. Call characteristics may include frequency of repeating a call,call content (e.g., requests or commands), or statistical similaritiesbetween the call and other call data. In some embodiments, APImanagement system 104 uses a call-classification model to determine acall relates to a malicious campaign.

In some embodiments, honeypot module 341 may be configured to provideinformation associated with a suspicious call to a client device (e.g.,client device 112) and/or an interface (e.g., interface 106). Honeypotmodule 341 may be configured to receive instructions in response toproviding the information. For example, honeypot module 341 may beconfigured to receive instructions to block a location associated withthe suspicious call, monitor activity, and/or route calls tonode-imitating models.

Honeypot module 341 may be configured to block a location. Blocking alocation may include rejecting calls associated with the location.Blocking a location may include maintaining a blacklist (a list ofblocked locations).

Honeypot module 341 may be configured to trace API calls back to thesource. This may involve tracking inbound IP calls, outbound IP calls,inbound transmission control protocol (TCP) calls, outbound TCP calls,inbound user datagram protocol (UDP) calls, and/or outbound UDP calls.

FIG. 4 depicts an exemplary process for training and implementing anode-testing model, consistent with disclosed embodiments. In someembodiments, API management system 104 performs process 400. One or moreof model-training module 336, node-testing module 337, translationmodule 338, routing module 339, dataset-clustering module 340, or othermodule of programs 335 may perform operations of process 400, consistentwith disclosed embodiments. It should be noted that other components ofsystem 100, including, for example, client device 112 or one or more APIsystems (e.g., API system 102 a, 102 b, 102 c) may perform one or moresteps of process 400.

Consistent with disclosed embodiments, steps of process 400 may beperformed on one or more cloud services using one or more ephemeralcontainer instances. For example, at any of the steps of process 400,API management system 104 may generate (spin up) an ephemeral containerinstance to execute a task, assign a task to an already-runningephemeral container instance (warm container instance), or terminate acontainer instance upon completion of a task. As one of skill in the artwill appreciate, steps of process 400 may be performed as part of anapplication interface (API) call.

At step 410, API node 102 receives calls (inputs) and produces APIoutputs based on the calls, consistent with disclosed embodiments. APInode 102 may be one of API systems 102 a, 102 b, 102 n, another APIsystem, an API function, an API subroutine, or an API node. API node 102may comprise sub-components (e.g., other API nodes, functions, routines,subroutines, or the like). At step 410, a call may be received from anycomponent of system 100 or a computing component outside system 100(e.g., via interface 106). For example, the call may be received fromanother API node, a client device, a routing module, an API managementsystem, and/or an interface. The call may include an API identifier. TheAPI node processes calls by performing one or more operations inresponse to the calls. API output may include routing information (e.g.,one or more API identifiers specifying a destination for the API output,or an instruction to return an output to a device). API output mayinclude an error message and/or an error code.

At step 420, API management system 104 trains a node-testing model 422,consistent with disclosed embodiments. In some embodiments, APImanagement system 104 may generate node-testing model 422 at step 420.As shown, node-testing model 422 may receive API input (an API call) andAPI output. API management system 104 may train node-testing model togenerate model output that matches the API output. In some embodiments,API management system 104 trains node-testing model 422 until one ormore training criteria are satisfied, consistent with disclosedembodiments. For example, a training criterion may be a percent matchbetween model output and API output for a number of API calls.

In some embodiments, model output includes a model result. The modelresult may include modeled API output (i.e., a possible result of an APIcall). Modeled API output may include data (e.g., a dataset, syntheticdata), descriptions of data (e.g., a data profile, a data schema, astatistical profile, and/or statistical metrics). Modeled API output mayinclude code to use in a call to an API and/or an identifier of an API.For example, modeled API output may include a call for a downstream nodeassociated with the identifier. In some embodiments, the model resultmay include an error message, a warning message, and/or error code. Themodel result may include a statement indicating the model result is anunknown result (i.e., the model has not been trained to produce anoutput based on the call).

In some embodiments, the model output includes a model-result category.The model-result category may indicate a confidence level associatedwith the model result (e.g., a likelihood that the model result matchesAPI output). The confidence level may include one or more probabilityvectors associated with particular model outputs based on the modelinput. The model-result category may indicate whether the model resultindicates a success (e.g., the model result includes data and noerrors), a possible warning (i.e., the model result includes a warningmessage), or failure (i.e., the model result includes an error message).As one of skill in the art will appreciate, the model-result categorymay include any kind of category associated with the call or the modelresult.

At step 430, API management system 104 implements node-testing model422. For example, API management system may route real-time API calls tonode-testing model 422 in real-time to process API calls and/or mayroute synthetic or stored API calls to node-testing model 422 (e.g., asdepicted in exemplary interface 250 of FIG. 2B). In some embodiments,model output includes a model result and a model-result category.

FIG. 5 depicts exemplary system 500 for training a translation model,consistent with the disclosed embodiments. As shown in FIG. 5, system500 includes a translation model 502, a node-testing model A 501 a, anda node-testing model B 501 b. Consistent with the disclosed embodiments,translation model 502 may be a translation model implemented bytranslation module 338. In some embodiments, node-testing model A 501 aand/or node-testing model B 501 b may be node-testing models 322,consistent with disclosed embodiments. Node-testing model A 501 a andnode-testing model B 501 b may be configured to generate model outputthat simulates the API output of API nodes including a specific versionof an API. These nodes may be remote from the translation model trainingsystem 500. Therefore, by using node-testing models, system 500 improvesefficiency because system 500 does not need to maintain or query the APInodes which the models simulate.

Translation model 502 may be trained to generate translated inputs frominputs, consistent with disclosed embodiments. Training translationmodel 522 may include passing inputs to node-testing model A 501 a toproduce model outputs A. Training may include passing translated inputsto node-testing model B 501 b to produce model output B. Training mayterminate when a comparison of model output B to model output Asatisfies a training criterion. As an example, the criterion may includewhether model output B matches model output A, or if model output Bmatches model output A for a threshold number of matches.

For example, during model training translation model 502 may receive aninput, which may include an API call, a dataset, an API response, amodel output from another node-testing model, or other input data, suchas metadata, identifiers, instructions, a source IP address, adestination IP address, or other additional data. Based on the input itreceives, translation model 502 determines an appropriate translatedinput, which may be sent to node-testing model B 501 b. Node-testingmodel B 501 b may then determine a new model output B, which may be sentto translation model 502.

Translation model 502 may compare the new model output B to an expectedmodel output. The expected model output may include API response, APIcall, metadata, or anything expected as the output of a system. In someembodiments, the expected model output may be associated with an outputexpected from a particular API. If translation model 502 determines thatthat the new model output B does not match the expected model output oris not within a predetermined range of tolerance associated with theexpected model output, then translation model 502 may determine that anew translated input should be produced, one that differs from thepreviously generated translated input. Translation model 502 may thendetermine a new translated input based on the new model output B and/orthe input and model output A, as well as any previously translatedinputs or outputs. If translation model 502 determines that the newmodel output B matches the expected model output, or is within apredetermined range of tolerance associated with the expected modeloutput, then translation model 502 may determine no adjustments areneeded, and it may continue to generate its most previous translatedinput.

In some embodiments, translation model 502 may repeatedly determinesuccessive translated inputs. These repeated translated inputs may bedetermined based on any inputs and/or outputs received by translationmodel 502. These inputs and outputs may be associated with current orprevious determinations of the translated input by translation model502. For example, translation model 502 may determine a new translatedinput based on a previously received input, a previously received modeloutput A, and a newly received model output B, which may have beenrecently determined by the most previous translated input.

FIG. 6 depicts exemplary process 600 for managing APIs, consistent withdisclosed embodiments. In some embodiments, API management system 104may perform process 600. One or more of model-training module 336,node-testing module 337, translation module 338, routing module 339,dataset-clustering module 340, and/or honeypot module 341, may performoperations of process 600, consistent with disclosed embodiments. Itshould be noted that other components of system 100 may perform one ormore steps of process 600, including, for example, API systems 102 a,102 b, 102 n and/or client device 112. Process 600 may be performedbased on user inputs or, in some embodiments, process 600 is executedautomatically by a program, script, or routine. For example, process 600may be performed according to a schedule or in response to a triggeringevent (e.g., following receipt of data at step 602).

At step 602, API management system 104 receives a request, consistentwith disclosed embodiments. API management system 104 may receive therequest from another component of system 100 or a computing componentoutside system 100 (e.g., via interface 106). The request may be arequest to test one or more API nodes, consistent with disclosedembodiments. The request may include one or more API calls. The requestmay include instructions to generate one or more synthetic API calls.The instructions may specify characteristics of the API calls. Forexample, the instructions may specify a range of API call parametersand/or a list of API functions. The request may include one or more APIidentifiers, routing tables, routing models, and/or informationidentifying a routing model. The request may include a request toprovide model output via a display (e.g., as depicted in interface 250of FIG. 2B) and/or to store model output.

At step 604, API management system 104 generates one or more API calls,consistent with disclosed embodiments. Generating an API call mayinclude generating the API call based on instructions received at step602. The API call may include an API identifier.

At step 606, API management system 104 identifies a testing model and/ora translation model, consistent with disclosed embodiments. Identifyinga node-testing model may be based on an API identifier. Identifying anode-testing model may be based on a received instruction (e.g., aninstruction received at step 602).

At step 608, API management system 104 translates the call, consistentwith disclosed embodiments. Translating the call may include identifyingan API version associated with the received call and implementing atranslation model to generate a translated call associated with anotherAPI version. In some embodiments, translating the call includesperforming steps of process 800, described in detail below.

At step 610, API management system 104 transmits the call to anode-testing model, consistent with disclosed embodiments. In someembodiments, transmitting the call at step 610 includes transmitting thegenerated call of step 604 and/or the translated call of step 608.

At step 612, API management system 104 receives node-testing modeloutput, consistent with disclosed embodiments. The node-testing modeloutput may include a model result and/or a model-result category.Following step 612, API management system 104 may perform any of steps614-620, alone or in combination.

At step 614, API management system 104 provides a notification,consistent with disclosed embodiments. The notification may include amodel output. The notification may include a model result and/or amodel-result category. In some embodiments, providing the notificationincludes displaying the notification at an interface (e.g., interface106). In some embodiments, providing the notification includestransmitting the notification to a client device (e.g., client device112). In some embodiments, providing the notification includes sendingthe notification to an enforcement system (e.g., enforcement system114).

At step 616, API management system 104 updates the call, consistent withdisclosed embodiments. Step 616 may include implementing a translationmodel. Updating the call may include characteristics of the API calls.For example, updating the call may include changing an API callparameter and/or an API function.

At step 618, API management system 104 updates the translation model,consistent with disclosed embodiments. In some embodiments, updating thetranslation model at step 618 is based on the model-result category. Forexample, the API management system 104 may transmit a translated call tothe node-testing model at step 610 and receive a model-result categoryat step 612 that indicates that the model result is a failure or anunknown result, trigging modeling training of the translation model.

At step 620, API management system 104 updates the node-testing model,consistent with disclosed embodiments. In some embodiments, updating thenode-testing model at step 620 is based on the model-result category.For example, the model-result category of step 612 may indicate that themodel result is a failure or an unknown result, trigging model trainingof the node-testing model.

It should be noted that process 600 may be performed iteratively. Forexample, following one of steps 614, 616, 618 and/or 620, API managementsystem 104 may again perform step 604 to generate another call based onnode-testing model output of step 612. As an illustrative example, APImanagement system 104 may perform process 600 for one or more arrows(API calls) depicted in exemplary interface 250 (FIG. 2B).

FIG. 7 depicts exemplary process 700 for training a node-testing model,consistent with disclosed embodiments. In some embodiments, APImanagement system 104 may perform process 700. In some embodiments,model-training module 336 performs process 700. One or more ofmodel-training module 336, node-testing module 337, translation module338, routing module 339, dataset-clustering module 340, and/or honeypotmodule 341, may perform operations of process 700, consistent withdisclosed embodiments. It should be noted that other components ofsystem 100 may perform one or more steps of process 700, including, forexample, API systems 102 a, 102 b, 102 n and/or client device 112.Process 700 may be performed based on user inputs or, in someembodiments, process 700 is executed automatically by a program, script,or routine. For example, process 700 may be performed according to aschedule or in response to a triggering event. Process 700 may beperformed by an ephemeral container instance, consistent with disclosedembodiments.

At step 702, model-training module 336 receives API call data,consistent with disclosed embodiments. API call data may include APIcall data API calls, API outputs, and/or API identifiers. Step 702 mayinclude receiving model characteristics and/or training criteria,consistent with disclosed embodiments. For example, received modelcharacteristics may include a model type, a model parameter, a modelhyperparameter, a desired outcome, belongingness to a model cluster,and/or belonginess of a model training dataset to a dataset cluster, thesimilarity of synthetic data generated by a model to actual data, orother characteristics.

At step 704, model-training module 336 generates a node-testing model,consistent with disclosed embodiments. The node-testing model mayinclude a machine learning model, consistent with disclosed embodiments.For example, the node-testing model may include one of an RNN, an LSTMmodel, or another neural network model. Generating the node-testingmodel at step 704 may include generating a plurality of model parameters(seeds) to use as starting points for model training. Generating thenode-testing model may include retrieving the node-testing model from adata storage (e.g., data 331 or model storage 110). Generating the modelmay be based on model characteristics received at step 702.

Generating the node-testing model at step 704 may be based on the APIcall data. For example, the API call data may include an API identifier,and generating the node-testing model may include retrieving anode-testing model previously trained to produce model output thatmatched API output of an API associated with the identifier.

At step 706, model-training module 336 trains the node-testing model,consistent with disclosed embodiments. Training the model at step 706may include training the node-testing model until one or more trainingcriteria are satisfied, consistent with disclosed embodiments. Forexample, a training criterion may be a percent match betweennode-testing model output and API output for a number of API calls.Training may be based on training criteria received at step 702.

At step 708, model-training module 336 provides the node-testing model,consistent with disclosed embodiments. Providing the model may includetransmitting the model to a module of API management system 104; storingthe model in a data storage (e.g., data 331, database 110, or other datastorage); displaying a graphical representation of the model (e.g., viainterface 106); and/or transmitting the model to another component ofsystem 100 (e.g. API system 102 a, 102 b, 102 n and/or client device112) and/or to a computing component outside system 100 (e.g., viainterface 106).

FIG. 8 depicts exemplary process 800 for testing a translation model,consistent with disclosed embodiments. In some embodiments, APImanagement system 104 performs process 800. One or more ofmodel-training module 336, node-testing module 337, translation module338, routing model module 339, dataset-clustering module 340, and/orhoneypot module 340 may perform operations of process 800, consistentwith disclosed embodiments. It should be noted that other components ofsystem 100, including, for example, client device 112, may perform oneor more steps of process 800.

Consistent with disclosed embodiments, steps of process 800 may beperformed on one or more cloud services using one or more ephemeralcontainer instances. For example, at any of the steps of process 800,API management system 104 may generate (spin up) an ephemeral containerinstance to execute a task, assign a task to an already-runningephemeral container instance (“warm container instance”), or terminate acontainer instance upon completion of a task. Steps of process 800 maybe performed using a system for training translation models, such assystem 500. As one of skill in the art will appreciate, steps of process800 may be performed before, after, or alongside an API request andresponse process. For example, process 800 may be initiated after APImanagement system 104 has become aware that a translation model hasproduced an output that does not correspond to an API version of adestination API node of the output. After becoming aware of thisinaccurate translation, API management system 104 may perform process800 to update a translation model so that accurate translation may takeplace, such that the output corresponds to the API version of thedestination API node of the output. In other cases, process 800 may beperformed following the release of a new API version, in order to keep atranslation model functioning effectively.

At step 802, API management system 104 receives input, consistent withdisclosed embodiments. The information may be received from, forexample, client device 112 and/or via interface 106. The input may havebeen sent from, for example, a device external to API management system104 in system 100, or from with API management system 104, such as fromanother module, which could be testing model module 337. The informationmay include an API call, API response, a translated input or an outputfrom translation model 502 and/or an API dataset. In some embodiments,the model is a machine learning model. The dataset may include real(actual) data and/or synthetic data, consistent with disclosedembodiments. In some embodiments, the information includes instructionsto generate a response to an API call and may include an API identifier,call arguments, or other parameters. In some embodiments, theinformation includes instructions to retrieve a model and/or a datasetfrom a data storage (e.g, data 231, model storage 108, and/or database110).

At step 804, API management system 104 selects one or more node-testingmodels, consistent with disclosed embodiments. A node-testing model maycorrespond to an API, consistent with disclosed embodiments. The versionof API may be associated with the call. For example, a first selectednode-testing model may correspond to the first version of the API, and asecond selected node-testing model may correspond to the second versionof the API. These node-testing models may be, for example, node-testingmodel A 501 a and/or node-testing model B 501 b.

Selecting a node-testing model at step 804 may include retrieving anode-testing model from a data storage (e.g., data 331, database 105).In some embodiments, the selected node-testing models are retrievedbased on instructions received at step 802. In some embodiments, step804 includes generating and/or training a node testing model, consistentwith disclosed embodiments.

At step 806, API management system 104 determines if the node-testingmodels generate model outputs that satisfy a test criterion, consistentwith disclosed embodiments. This test criterion may include, forexample, a percent match between a model output and an expected modeloutput, or a percent match between a schema of the model output and anexpected schema. Determining whether a model output satisfies a testcriterion related to a schema may involve using a scheme validationframework, such as Marshmellow, a scheme validation framework that canbe implemented using Python. The expected output may include a previousmodel output generated by API management system 104, and/or a range ofdesirable model outputs. For example, the expected output may includethe model output from the node-testing model corresponding to a firstversion of an API (e.g., model output A, shown in FIG. 5). In someembodiments, the expected output may include system logs or analyticdata. In some embodiments, the logs or analytic data may be unrelated tothe input. For example, honeypot module 341 may generate synthetic data,and the logs may be unrelated to the input. In some embodiments, thetest criterion includes a determination of whether a distribution offoreign keys and/or data values is within an expected range. Forexample, the test criterion may be based on a data profile or astatistic metric of API calls, consistent with disclosed embodiments.API management system 104 may also determine if the node-testing modelsgenerated model outputs that satisfy a test criterion based on whetheran error message is received. In some embodiments, determining whether amodel output satisfies a testing criterion may involve determiningwhether multiple model outputs satisfy a desired distribution related toexpected model outputs.

At step 807 API management system 104 sends a notification, consistentwith disclosed embodiments. In some embodiments, the notification may besent via interface 106. In some embodiments, sending the notificationincludes sending the notification to client device 112. In someembodiments, sending the notification includes displaying thenotification at a display of I/O devices 320. The notification mayinclude a warning of changes at an API node, an indication that changesat an API node are breaking changes, an indication of changes detected,and/or possible recommended actions. Recommended actions may include,for example, changing an API version at a node, providing a manualtranslation, and/or conducting supervised training of a translationmodel. As shown, step 807 may follow at least one of step 806 and/orstep 814 following a determination that a node-testing model output doesnot satisfy a test criterion.

At step 808, API management system 104 selects a translation model basedon the input, consistent with disclosed embodiments. For example, theinput may be an API call and the version of the API may be determined.Based on the information associated with the input, an appropriatetranslation model may be selected. In some embodiments, the translationmodel may be translation model 502. The translation model may beconfigured to translate an input, such as an API call, from a firstversion of an API to a second version of an API. For example, the firstversion of an API may be the version currently associated with theinput, and the second API version may be a version associated with adestination API node. The translation model may be selected using atable that associates APIs and API versions with correspondingtranslation models, or using a versioning model that associates APIs andtheir versions with corresponding models. For example, if the input isan API call of a particular version in need of translation, atranslation model for translating calls of that version may be selectedfor testing.

Selecting a translation model at step 808 may include retrieving anode-testing model from a data storage (e.g., data 331, database 105).In some embodiments, the selected translation model may be retrievedbased on instructions received at step 802. In some embodiments, step808 includes generating and/or training a translation model, consistentwith disclosed embodiments. Training the translation model may includetraining the model using API call data associated with a first versionof an API and API call data associated a second version of an API,consistent with disclosed embodiments. The API call data may beretrieved from a data storage (e.g., data 331, database 105).

At step 810, API management system 104 generates a translated inputbased on the received input, using the translation model, consistentwith disclosed embodiments. In some embodiments, the translated inputmay be based on information received at step 802, which may includeother inputs or outputs. These may include API calls, API datasets, orother data related to an API.

At step 812, API management system 104 generates one or more modeloutputs of the selected node-testing models, consistent with disclosedembodiments. In some embodiments, at least one model output may begenerated by a node-testing model corresponding to a second version ofan API based on the translated input (e.g., as illustrated in modeloutput B of FIG. 5). In some embodiments, at least one model output maybe generated by a node-testing model corresponding to a second versionof an API based on the input (e.g., as illustrated in model output A ofFIG. 5).

At step 814, API management system 104 determines if the model outputgenerated at step 812 satisfies a test criterion, consistent withdisclosed embodiments. This test criterion may include, for example, apercent match between a model output corresponding to a second versionof an API generated at step 812 and an expected model output, or apercent match between a schema of the model output and an expectedschema. In some embodiments, the test criterion includes a determinationof whether a distribution of foreign keys and/or data values is withinan expected range. For example, the test criterion may be based on adata profile or a statistic metric of API calls, consistent withdisclosed embodiments. The expected model output may include a previousoutput generated by API management system 104, and/or a range ofdesirable outputs. For example, the expected model output may includethe model output from the node-testing model corresponding to a firstversion of an API (e.g., model output A, shown in FIG. 5). In anotherexample, API management system 104 may see if a key or value of themodel output is within a desirable range. In some embodiments, theexpected output may include system logs or analytic data. In someembodiments, the logs or analytic data may be unrelated to the input.API management system 104 may also determine if the output generated atstep 812 meets a test criterion based on whether an error message isreceived. In some embodiments, API management system 104 may determineif the model output generated at step 812 meets a test criterion byanalyzing how well the model output corresponds to a particular versionof an API, which may be associated with an API node that will receive anAPI output based on the model output. For example, API management system104 may determine whether the model output has a schema that accordswith a particular version of an API, and/or whether the model outputwill cause an error at a destination node. API management system 104 maydetermine that no model training is needed if the model output accordswith a particular version of an API, or if it does not accord with thatversion but will not cause an error at a destination node. In someembodiments, determining whether a model output satisfies a testingcriterion may involve determining whether multiple model outputs satisfya desired distribution related to expected model outputs.

At step 816, API management system 104 trains the translation modelbased on the model output generated at step 812, consistent withdisclosed embodiments. In some embodiments, the training may occur ifthe model output generated at step 812 does not meet the trainingcriterion. Testing may involve an iterative test whereby steps 802-814are repeated successively, which may involve changing the inputs forsome iterations. This iterative test may involve API management system104 changing rules associated with a translation module 338 thatimplements the translation model. Repeating steps 802-814 may generatemodel inputs and/or model outputs that are used only for trainingpurposes, which may be termed intermediate model inputs and intermediatemodel outputs. Steps 802-814 may be repeated until API management system104 determines that the model output generated at step 812 satisfies atraining criterion. In some embodiments, multiple tests may be runsimultaneously. In some embodiments, training the translation model mayinclude producing an updated translation model, which may operate usinga translation module 338 that has new rules.

Training the translation model may also trigger API management system104 to train or generate another translation model. For example, afterupdating a translation model that translates API calls from a firstversion to a second version, process 800 may be used to train atranslation model that translates API calls from a second version to afirst version. Training a translation model may also trigger APImanagement system 104 to train another model, such as one implemented bymodel training module 336, translation module 338, or routing modelmodule 339. These other models could include a node-testing model. Forexample, API management system 104 may update node-testing models thatwere used to train a translation model when the node-testing modelproduces an error, an unknown result, or a result that does not satisfya training criterion during translation model training.

At step 818, API management system 104 may provide an updatedtranslation model trained at step 816, consistent with the disclosedembodiments. This may include transmitting the pre-updated model and/orupdated model to a module of API management system 104; storing thepre-updated model and/or updated model in a data storage (e.g., data331, database 110, or other data storage); displaying a graphicalrepresentation of the model (e.g., via interface 106); and/ortransmitting the pre-updated and/or updated model to another componentof system 100 (e.g. API system 102 a, 102 b, 102 n and/or client device112) and/or to a computing component outside system 100 (e.g., viainterface 106). In some embodiments, any pre-updated instances of amodel may be discarded after a model is updated.

At step 820, API management system 104 may provide a node-testing modeloutput generated at step 812, consistent with the disclosed embodiments.This may include transmitting the node-testing model output to anothertranslation model 502 and/or a module of API management system 104;storing the node-testing model output in a data storage (e.g., data 331,database 110, or other data storage); displaying a graphicalrepresentation of the node-testing model output (e.g., via interface106); and/or transmitting the node-testing model output to anothercomponent of system 100 (e.g. API system 102 a, 102 b, 102 n and/orclient device 112) and/or to a computing component outside system 100(e.g., via interface 106). In some embodiments, any pre-updatedinstances of a model may be discarded after a model is updated.

FIG. 9 is a depicts exemplary process 900 for translating an input,consistent with disclosed embodiments. In some embodiments, APImanagement system 104 performs process 900. One or more ofmodel-training module 336, node-testing module 337, translation module338, routing module 339, dataset-clustering module 340, and/or honeypotmodule 341, may perform operations of process 700, consistent withdisclosed embodiments. It should be noted that other components ofsystem 100, including, for example, client device 112, may perform oneor more steps of process 900.

Consistent with disclosed embodiments, steps of process 900 may beperformed on one or more cloud services using one or more ephemeralcontainer instances. For example, at any of the steps of process 900,API management system 104 may generate (spin up) an ephemeral containerinstance to execute a task, assign a task to an already-runningephemeral container instance (“warm container instance”), or terminate acontainer instance upon completion of a task. As one of skill in the artwill appreciate, steps of process 900 may be performed as part ofprocessing an API call.

At step 902, API management system 104 receives an API call, consistentwith disclosed embodiments. This API call may be received frommodel-training module 336, testing model module 337, translation modelmodule 338, routing model module 339, dataset-clustering module 340,honeypot module 341, or other components of system 100, including, forexample, client device 112. API management system 104 may receive otherinput with the API call, such as an API dataset, API response, modeloutput from another node-testing model, or other input data, such asmetadata, identifiers, instructions, system logs, analytic data, orother additional data.

At step 904, API management system 104 determines the API version of theAPI call received at step 902, consistent with disclosed embodiments.This determination may be based on an API identifier contained in theAPI call. After determining the API version, API management system 104may determine an appropriate model to use for translating the API call.The model may be selected using a table that associates APIs and APIversions with corresponding models, or using a separate model thatassociates APIs and their versions with corresponding models, as in step804.

At step 906, API management system 104 translates the API call. In someembodiments, translating the API call includes using a translation modelas described in translation module 338. For example, if the API call isassociated with a first version of an API and it has a destination nodewith a second version of an API, API management system 104 may translatethe API call from the first version to the second version.

At step 908, API management system 104 transmits the translated APIcall, consistent with disclosed embodiments. Transmitting the translatedAPI call may include transmitting the translated API call to an APInode. In some embodiments, API management system 104 may transmit thetranslated API call to any of programs 335, to another part of APImanagement system 104, or to another component of system 100.Transmitting the translated call may include storing the received and/ortranslated call in a data storage (e.g., database 110, data 331). Insome embodiments, calls, responses, and other API inputs and outputs maybe stored as part of a dataset, which may be stored in a data storage(e.g., database 110, data 331). Such datasets may be used to trainmodels, for example, by using process 700.

At step 910, API management system 104 receives an API output from anAPI node, consistent with disclosed embodiments. The API output may havebeen generated based on the translated API. For example, the API outputmay include a result generated by an API in response to the translatedAPI call transmitted by API management system 104.

At step 912, API management system 104 determines an API versionassociated with a destination API node, consistent with disclosedembodiments. The destination API node may be the next node to which theoutput will be transmitted, which may be the same API node from whichthe API call was received at step 902 (i.e., a source node). In someembodiments, API management system 104 may determine that the APIversion of the API output is not the same as the API version of the as adestination API node, and that this incompatibility may produce an errorat an API node. The API version of the destination API node may bedetermined from information contained in the API call. In someembodiments, the API version of the destination API node may be the sameas the API version of the API call determined at step 904.

At step 914, API management system 104 translates the API output,consistent with disclosed embodiments. For example, API managementsystem 104 may translate an API output configured for one version of anAPI to an API output configured for a different version of an API. Thedifferent version may be one used by a node that may receive the APIoutput.

At step 916, API management system 104 transmits the translated APIoutput, consistent with disclosed embodiments. In some embodiments, thetranslated API output may be transmitted to the API node from which theAPI call was received at step 902. For example, the source API node maybe part of a user device that sent the API call at step 902. In someembodiments, the translated API output may be transmitted to another APInode. In some embodiments, the translated API output may be transmittedto another part of API management system 104, another component ofsystem 100 (e.g., one of API systems 102 a, 102 b, 102 c), and/or or acomputing component outside system 100 (e.g., via interface 106).

FIG. 10 depicts exemplary process 1000 for training and implementing anode-imitating model, consistent with disclosed embodiments. In someembodiments, API management system 104 performs process 1000. One ormore of model-training module 336, node-testing module 337, translationmodule 338, routing module 339, dataset-clustering module 340, and/orhoneypot module 341 may perform operations of process 1000, consistentwith disclosed embodiments. Process 1000 may be performed based on userinputs or, in some embodiments, process 1000 is executed automaticallyby a program, script, or routine. For example, process 1000 may beperformed according to a schedule or in response to a triggering event.It should be noted that other components of system 100, including, forexample, client device 112 or one or more API systems (e.g., API system102 a, 102 b, 102 c) may perform one or more steps of process 1000.

Consistent with disclosed embodiments, steps of process 1000 may beperformed on one or more cloud services using one or more ephemeralcontainer instances. For example, at any of the steps of process 1000,API management system 104 may generate (spin up) an ephemeral containerinstance to execute a task, assign a task to an already-runningephemeral container instance (warm container instance), or terminate acontainer instance upon completion of a task. As one of skill in the artwill appreciate, steps of process 1000 may be performed as part of anapplication interface (API) call.

At step 1010, API node 102 receives calls (inputs) and produces APIoutputs based on the calls as previously described in relation to step410, consistent with disclosed embodiments.

At step 1020, API management system 104 trains a node-imitating model1022, consistent with disclosed embodiments. In some embodiments, APImanagement system 104 may generate node-imitating model 1022 at step1020. Generating and/or training node-imitating model 1022 may be basedon data received or retrieved from model storage 108, including one ormore models, model characteristics, and/or training criteria.Node-imitating model 1022 may be configured to retrieve and/or receivedata from database 110, including one or more datasets. In someembodiments, training node-imitating model 1022 may include receiving anode-testing model (e.g., node-testing model 422) from a data storage(e.g., model storage 108) and training the node-testing model.

API management system 104 may train node-imitating model 1022 togenerate model output based on the API output. In some embodiments,training node-imitating model 1022 to generate model output includetraining the model to generate synthetic data. The synthetic data maysatisfy a similarity metric when compared to the API output. In someembodiments, model output of node-imitating model 1022 may include adata marker, consistent with disclosed embodiments.

API management system 104 may train node-imitating model 1022 togenerate model output having a predetermined data profile. For example,the predetermined data profile may be based on one or more data markers(e.g., node-imitating model 1022 may generate data with two data markershaving a predetermined covariance between data columns).

At step 1030, API management system 104 implements node-imitating model1022. For example, API management system may route real-time API callsto node-imitating model 1022 to process API calls.

FIG. 11, depicts exemplary process 1100 for managing unauthorized APIcalls, consistent with disclosed embodiments. In some embodiments, APImanagement system 104 may perform process 1100. One or more ofmodel-training module 336, node-testing module 337, translation module338, routing module 339, dataset-clustering module 340, and/or honeypotmodule 341, may perform operations of process 1100, consistent withdisclosed embodiments. It should be noted that other components ofsystem 100 may perform one or more steps of process 1100, including, forexample, API systems 102 a, 102 b, 102 n and/or client device 112.Process 1100 may be performed based on user inputs or, in someembodiments, process 1100 is executed automatically by a program,script, or routine. For example, process 1100 may be performed accordingto a schedule or in response to a triggering event. Process 1100 may beperformed by an ephemeral container instance, consistent with disclosedembodiments.

At step 1102, API management system 104 receives a call, consistent withdisclosed embodiments. Receiving the call may include receiving the callfrom a client device (e.g., client device 112) or a computing componentoutside system 100. Receiving a call may include receiving the call froman API node (e.g., one of API systems 102 a, 102 b, or 102 n, or acomponent of one of the API systems). For example, the call may be acall between nodes of different routing layers as depicted in exemplaryinterface 250 of FIG. 2B.

At step 1104, API management system 104 classifies the call, consistentwith disclosed embodiments. Classifying the call may include using acall-classification model, as described above. In some embodiments, thecall may be classified as one of authorized, suspicious, unauthorized,or another call class. Classifying the call may be based on a useraccount, a log, a failed authentication attempt, a packet sniffingevent, a rate of pinging, an IP address, and/or a MAC address.

At step 1106, API management system 104 determines whether the callrelates to a malicious campaign, consistent with disclosed embodiments.In some embodiments, determining that the call relates to a maliciouscampaign includes determining that call characteristics are similar toone or more suspicious calls and/or one or more unauthorized calls. Insome embodiments, API management system 104 uses a call-classificationmodel to determine a call relates to a malicious campaign.

At step 1108, API management system 104 sends the call to anode-imitating model, consistent with disclosed embodiments. In someembodiments, step 1108 includes generating or training a node-imitatingmodel, consistent with disclosed embodiments. It should be noted that,before or after step 1108, API management system 104 may continue toroute legitimate or authorized call to API nodes or may generate new APInodes (e.g., a secondary live system) to manage legitimate API calls.

At step 1110, API management system 104 receives model output of thenode-imitating model, consistent with disclosed embodiments. Aspreviously described in reference to honeypot module 341, model outputof the node-imitating model may include synthetic data based on APIoutput.

At step 1112, API management system 104 transmits model output and/orinformation based on the model output, consistent with disclosedembodiments. Transmitting at step 1112 may include transmitting to alocation associated with the call. Transmitting at step 1112 may includetransmitting the model output or information based on the model outputto a second node-imitating model. For example, if the model outputincludes a call to a downstream API node, step 1112 may includerepeating steps 1108 and 1110. Steps 1108 through 1112 may be repeatedany number of times. Repeating steps 1108 through 1112 may includeoperations performed by routing module 339 (e.g., identifying downstreamAPI nodes associated with an API call, using a routing table, and/orusing a routing model).

At step 1114, API management system 104 identifies a location associatedwith the call, consistent with disclosed embodiments. The location mayinclude an account, an IP address, a MAC address, a URL, or the like.

At step 1116, API management system 104 monitors activity, consistentwith disclosed embodiments. Monitoring activity may include classifyingfuture calls (e.g., using a call-classification model), routing callsassociated with the location to a node-imitating model, and/or comparingnew calls to the call received at step 1102. Step 1116 may includedetermining, based on the comparison, that a new call is relates to thecall receive at step 1102.

At step 1118, API management system 104 generates a log, consistent withdisclosed embodiments. In some embodiments, the log includes informationassociated with the received call (e.g., the location; a timestamp; aclassification as authorized, unauthorized, or suspicious). In someembodiments, the log includes information associated with the monitoringof step 1116.

At step 1120, API management system 104 provides the log, consistentwith disclosed embodiments. Providing the log may include displaying thelog in an interface (e.g., interface 106). Providing the log may includetransmitting the log to another component of system 100 (e.g., clientdevice 112; enforcement system 114) and/or or a computing componentoutside system 100 (e.g., via interface 106).

At step 1122, API management system 104 blocks the location, consistentwith disclosed embodiments. Blocking the location may include updating ablacklist.

FIG. 12 depicts exemplary process 1200 for training a node-imitatingmodel, consistent with disclosed embodiments. In some embodiments, APImanagement system 104 may perform process 1200. One or more ofmodel-training module 336, node-testing module 337, translation module338, routing module 339, dataset-clustering module 340, and/or honeypotmodule 341, may perform operations of process 1200, consistent withdisclosed embodiments. It should be noted that other components ofsystem 100 may perform one or more steps of process 1200, including, forexample, API systems 102 a, 102 b, 102 n and/or client device 112.Process 1200 may be performed based on user inputs or, in someembodiments, process 1200 is executed automatically by a program,script, or routine. For example, process 1200 may be performed accordingto a schedule or in response to a triggering event. Process 1200 may beperformed by an ephemeral container instance, consistent with disclosedembodiments.

At step 1202, model-training module 336 receives API call data,consistent with disclosed embodiments. API call data may include APIcall data API calls, API outputs, and/or API identifiers. Step 1202 mayinclude receiving model characteristics and/or training criteria,consistent with disclosed embodiments. For example, received modelcharacteristics may include a model type, a model parameter, a modelhyperparameter, a desired outcome, belongingness to a model cluster,and/or belonginess of a model training dataset to a dataset cluster, thesimilarity of synthetic data generated by a model to actual data, orother characteristics.

At step 1204, model-training module 336 generates a node-implementingmodel, consistent with disclosed embodiments. The node-implementingmodel may include a machine-learning model, consistent with disclosedembodiments. For example, the node-implementing model may include one ofan RNN, an LSTM model, a seq2seq model, a CNN, a GAN, an autoencoder, oranother neural network model. Generating the node-implementing model atstep 1204 may include generating a plurality of model parameters (seeds)to use as starting points for model training. Generating thenode-implementing model may include retrieving a node-testing model or anode-implementing model from a data storage (e.g., data 331 or modelstorage 110). Generating the node-implementing model may be based onmodel characteristics received at step 1202.

Generating the node-implementing model at step 1204 may be based on theAPI call data. For example, the API call data may include an APIidentifier, and generating the node-implementing model may includeretrieving a node-implementing model previously trained to produce modeloutput that matched API output of an API associated with the identifier.

At step 1206, model-training module 336 trains the node-implementingmodel, consistent with disclosed embodiments. Training the model at step1206 may include training the node-implementing model until one or moretraining criteria are satisfied, consistent with disclosed embodiments.Training may be based on training criteria received at step 1202. Insome embodiments, training includes training the node-implementing modelto generate synthetic data. The synthetic data may satisfy a similaritymetric when compared to the API output. In some embodiments, modeloutput of the node-imitating model may include a data marker, consistentwith disclosed embodiments

At step 1208, model-training module 336 provides the node-implementingmodel, consistent with disclosed embodiments. Providing the model mayinclude transmitting the model to a module of API management system 104;storing the model in a data storage (e.g., data 331, database 110, orother data storage); displaying a graphical representation of the model(e.g., via interface 106); and/or transmitting the model to anothercomponent of system 100 (e.g. API system 102 a, 102 b, 102 n and/orclient device 112) and/or to a computing component outside system 100(e.g., via interface 106).

FIG. 13 depicts exemplary process 1300 for identifying suspicious data,consistent with disclosed embodiments. In some embodiments, APImanagement system 104 may perform process 1300. One or more ofmodel-training module 336, node-testing module 337, translation module338, routing module 339, dataset-clustering module 340, and/or honeypotmodule 341, may perform operations of process 1300, consistent withdisclosed embodiments. It should be noted that other components ofsystem 100 may perform one or more steps of process 1300, including, forexample, API systems 102 a, 102 b, 102 n and/or client device 132.Process 1300 may be performed based on user inputs or, in someembodiments, process 1300 is executed automatically by a program,script, or routine. For example, process 1300 may be performed accordingto a schedule or in response to a triggering event. Process 1300 may beperformed by an ephemeral container instance, consistent with disclosedembodiments.

At step 1302, API management system 104 searches a remote computingresource, consistent with disclosed embodiments. Searching the remotecomputer resource may include searching a network (e.g., a virtualprivate network, the internet, or any other network). Searching theremote computing resource may include searching a database. Searchingthe remote computing resource may include searching enforcement system114. Searching the remote computing resource at step 1302 may includesearching for a data marker. Searching the remote computing resource atstep 1302 may include searching for a dataset

At step 1304 API management system 104 receives suspicious data,consistent with disclosed embodiments. The suspicious data may include adataset. In some embodiments, the suspicious data may be received in arequest from a client device 112 or enforcement system 114. In someembodiments, the suspicious data may be received in response to thesearching (e.g., as a download, a search result, or a filetransmission).

At step 1306, API management system 104 retrieves model output from oneor more node-imitating models, consistent with disclosed embodiments.Retrieving model output may include retrieving a data marker and/or adata profile.

At step 1308, API management system 104 determines a suspiciousnessscore of the data, consistent with disclosed embodiments. In someembodiments, the suspiciousness score is a likelihood that the dataderives from API calls to an API node (e.g. API systems 102 a, 102 b,102 n or their components) and/or API management system 104. In someembodiments, the suspiciousness score is based on detecting a datamarker in the suspicious data. For example, the suspiciousness score maybe based on a frequency of the data marker in a dataset. The data markermay be a data marker retrieved at step 1306.

In some embodiments, the suspiciousness score is based on a data profileof the suspicious data. For example, step 1308 may include determining asimilarity metric of a data profile of a dataset of the suspicious dataand a data profile of a stored dataset (or a stored data profile). Thesuspiciousness score may be based on the similarity metric (e.g., if thesimilarity metric indicates high similarity, the suspiciousness scoremay be high). The data profile of a dataset of the suspicious data maybe received (e.g. at step 1304) or determined by API management system104 (as described above). The stored dataset and/or stored data profilemay be stored, for example, in one of data 331 or database 110. Thestored dataset and/or stored data profile may be retrieved at step 1306.

At step 1310, API management system 104 identifies a location associatedwith the suspicious data, consistent with disclosed embodiments. Thelocation may be received at step 1304. The location may be identifiedbased on a call tracing.

At step 1312, API management system 104 provides the suspiciousnessscore or location, consistent with disclosed embodiments. Providing thesuspiciousness score may include displaying the suspiciousness score inan interface (e.g., interface 106). Providing the suspiciousness scoremay include transmitting the suspiciousness score to another componentof system 100 (e.g., client device 112; enforcement system 114) and/oror a computing component outside system 100 (e.g., via interface 106).

Systems and methods disclosed herein involve unconventional improvementsover conventional approaches to managing APIs. Descriptions of thedisclosed embodiments are not exhaustive and are not limited to theprecise forms or embodiments disclosed. Modifications and adaptations ofthe embodiments will be apparent from consideration of the specificationand practice of the disclosed embodiments. Additionally, the disclosedembodiments are not limited to the examples discussed herein. It shouldbe noted that client device 112 and/or one or more of API system 102 a,102 b, 102 n may perform any of the features or steps described above inregard to API management system 104 in reference to the variousembodiments and processes.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments. For example, the describedimplementations include hardware and software, but systems and methodsconsistent with the present disclosure can be implemented as hardwarealone.

Computer programs based on the written description and methods of thisspecification are within the skill of a software developer. The variousfunctions, scripts, programs, or modules can be created using a varietyof programming techniques. For example, programs, scripts, functions,program sections or program modules can be designed in or by means oflanguages, including JAVASCRIPT, C, C++, JAVA, PHP, PYTHON, RUBY, PERL,BASH, or other programming or scripting languages. One or more of suchsoftware sections or modules can be integrated into a computer system,non-transitory computer-readable media, or existing communicationssoftware. The programs, modules, or code can also be implemented orreplicated as firmware or circuit logic.

Moreover, while illustrative embodiments have been described herein, thescope includes any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication, which examples are to be construed as non-exclusive.Further, the steps of the disclosed methods can be modified in anymanner, including by reordering steps or inserting or deleting steps. Itis intended, therefore, that the specification and examples beconsidered as exemplary only, with a true scope and spirit beingindicated by the following claims and their full scope of equivalents.

What is claimed is:
 1. A system for managing application programminginterfaces (APIs), comprising: one or more memory units storinginstructions; and one or more processors configured to execute theinstructions to perform operations comprising: training a node-testingmodel corresponding to an API node based on API call data and a trainingcriterion; generating, by the node-testing model, a model result, themodel result comprising modeled API output; determining an estimate of alikelihood that the model result matches a result of the API node; andupdating a translation model associated with the API node based on themodel result and the estimate.
 2. The system of claim 1, wherein theestimate of the likelihood that the model result matches the result ofthe API node comprises a likelihood that the model result contains anerror or warning.
 3. The system of claim 1, wherein the operationsfurther comprise: training the node-testing model by determining that atraining modeled API output of the node-testing model does not satisfythe training criterion; and based on the determination that the trainingmodeled output of the node-testing model does not satisfy the trainingcriterion, further training the node-testing model.
 4. The system ofclaim 1, wherein the modeled API output comprises one or more of: adataset, synthetic data, a data profile, a data schema, a statisticalprofile, statistical metrics, an identifier of an API, and a code to usein a call to an API.
 5. The system of claim 1, wherein the estimate ofthe likelihood that the model result matches a result of the API nodeincludes a probability vector associated with the model result.
 6. Thesystem of claim 1, wherein: the training criterion is associated with aschema of the model result from the node-testing model and an expectedschema; and determining that the model result from the node-testingmodel satisfies the training criterion is based on a schema validationframework.
 7. The system of claim 1, wherein the node-testing model is afirst node-testing model and the translation model is (i) updated basedon the model result from the first node-testing model and a model resultfrom a second node-testing model and (ii) trained to translate an inputto a translated input, the input and the translated input associatedwith different APIs.
 8. A method for managing application programminginterfaces (APIs), comprising: training a node-testing modelcorresponding to an API node based on API call data and a trainingcriterion; generating, by the node-testing model, a model result, themodel result comprising modeled API output; determining an estimate of alikelihood that the model result matches a result of the API node; andupdating a translation model associated with the API node based on themodel result and the estimate.
 9. The method of claim 8, wherein theestimate of the likelihood that the model result matches the result ofthe API node comprises a likelihood that the model result contains anerror or warning.
 10. The method of claim 8, further comprising:training the node-testing model by determining that a training modeledAPI output of the node-testing model does not satisfy the trainingcriterion; and based on the determination that the training modeledoutput of the node-testing model does not satisfy the trainingcriterion, further training the node-testing model.
 11. The method ofclaim 8, wherein the modeled API output comprises one or more of: adataset, synthetic data, a data profile, a data schema, a statisticalprofile, statistical metrics, an identifier of an API, and a code to usein a call to an API.
 12. The method of claim 8, wherein the estimate ofthe likelihood that the model result matches a result of the API nodeincludes a probability vector associated with the model result.
 13. Themethod of claim 8, wherein: the training criterion is associated with aschema of the model result from the node-testing model and an expectedschema; and determining that the model result from the node-testingmodel satisfies the training criterion is based on a schema validationframework.
 14. The method of claim 8, wherein the node-testing model isa first node-testing model and the translation model is (i) updatedbased on the model result from the first node-testing model and a modelresult from a second node-testing model and (ii) trained to translate aninput to a translated input, the input and the translated inputassociated with different APIs.
 15. A non-transitory computer readablemedium having instructions thereon, the instructions when executed by acomputer, causing the computer to manage application programminginterfaces (APIs), the instructions causing operations comprising:training a node-testing model corresponding to an API node based on APIcall data and a training criterion; generating, by the node-testingmodel, a model result, the model result comprising modeled API output;determining an estimate of a likelihood that the model result matches aresult of the API node; and updating a translation model associated withthe API node based on the model result and the estimate.
 16. The mediumof claim 15, wherein the estimate of the likelihood that the modelresult matches the result of the API node comprises a likelihood thatthe model result contains an error or warning.
 17. The medium of claim15, the operations further comprising: training the node-testing modelby determining that a training modeled API output of the node-testingmodel does not satisfy the training criterion; and based on thedetermination that the training modeled output of the node-testing modeldoes not satisfy the training criterion, further training thenode-testing model.
 18. The medium of claim 15, wherein the modeled APIoutput comprises one or more of: a dataset, synthetic data, a dataprofile, a data schema, a statistical profile, statistical metrics, anidentifier of an API, and a code to use in a call to an API.
 19. Themedium of claim 15, wherein the estimate of the likelihood that themodel result matches a result of the API node includes a probabilityvector associated with the model result.
 20. The medium of claim 15,wherein: the training criterion is associated with a schema of the modelresult from the node-testing model and an expected schema; anddetermining that the model result from the node-testing model satisfiesthe training criterion is based on a schema validation framework.